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(57) An image processing method and program 
comprises : acquiring (S1) an object image; displaying 
(S2) the object image acquired by the image acquiring 
element; capturing a predetermined image pattern from 
within the object image acquired by the image acquiring 
element in order to recognize (S3) position information 
about the image pattern relative to the object image; dis- 
playing (S6) a menu image in overlaid relation to the dis- 
played object image in accordance with an outcome of 
the recognition by the captured position information rec- 
ognizing element; and starting (S7) a previously related 
process based on the displayed menu image and rec- 
ognized on the position information. Image processing 
apparatus and a program storage medium are also dis- 
closed. 
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Description 

BACKGROUND OF THE INVENTION 

[0001] Die present invention relates to: an image 5 
processing apparatus; an image processing method; an 
image processing program storage medium; and a com- 
puter program product for facilitating diverse input op- 
erations such as selection of a menu and starting of an 
application program. 10 
[0002] Recent years have witnessed widespread use 
of personal computer operating systems such as Win- 
dows 95 (registered trademark) and Windows 98 (reg- 
istered trademark), both from Microsoft Corporation in 
the United States. Such operating systems offer on a is 
display screen various task bars with indications visible 
to users such as buttons representing applications each 
having an active window 

[0003] A technique dealing with the use of the task 
bar is disclosed in detail by Japanese Published Unex- 20 
amined Patent Application No. Hei 8-255066. Below is 
an excerpt from the cited patent application. 
[0004] Illustratively, the task bar includes buttons that 
inform a user which tasks are currently active. The task 
bar may also include a start menu button comprising 25 
menu items that allow the user to activate desired pro- 
grams, open documents and gain access to a menu of 
system settings. 

[0005] In a typical computer system including a cen- 
tral processing unit (CPU) and such peripherals as a 30 
mouse, a keyboard and a video display, the start menu 
button on the task bar is used illustratively as follows: 
the start menu button is operated by the user to open 
the start menu for gaining access to programs, docu- 
ments, system controls and help information. Initially, 3S 
the user points a mouse cursor to the start menu button 
and clicks a left mouse button. The actions cause the 
start menu to appear. 

[0006] The start menu typically includes such menu 
items as programs, search, setting and help. The "pro- 40 
grams" menu item when selected allows the user to ac- 
cess a program menu displayed hierarchically from the 
start menu. The program menu displays a plurality of 
application programs and a group of programs that may 
be selected by the user. 45 
[0007] The trouble is that the user is required repeat- 
edly to perform complicated actions when selecting and 
activating a desired application program from the task 
bar comprising the start menu button. 
[0008] Specifically, the user first points the mouse cur- so 
sor to the start menu button on the task bar and clicks 
the left mouse button to display the start menu. Next, 
the user points the mouse cursor to the "programs" 
menu item in the start menu and clicks the left mouse 
button to display the program menu. The user then ss 
points the mouse cursor to an indication of the desired 
application program in the program menu, and again 
clicks the left mouse button. Another series of point and 



click actions is needed if there is an application program 
group. Finally the CPU activates the selected applica- 
tion program. 

[0009] The user is required conventionally to repeat 
such complicated actions before starting the desired ap- 
plication program. In particular, if the user is already 
handling some other chores with one hand, operating 
the pointing device with the other hand often turns out 
to be a difficult task from a user interface point of view. 
The same difficulty is encountered when other menu 
items such as "help" are clicked from the start menu pro- 
gram for desired processing. 

[0010] Meanwhile, the so-called bar code system is 
used extensively today in various industrial fields. The 
system involves encoding in bars alphanumeric charac- 
ters representing identification numbers and other infor- 
mation about goods and articles, and attaching such 
codes to the goods and products so that their identifica- 
tion information may later be scanned and retrieved by 
an optical recognition apparatus called a barcode scan- 
ner. The bar code system, used primarily at cash regis- 
ters or like equipment in retail stores, allows bar codes 
on sold products to be optically scanned so as to acquire 
their identification numbers. The acquired identification 
numbers are used as a basis for retrieving previously 
stored prices of the corresponding products from a da- 
tabase, and the retrieved prices are displayed on a dis- 
play part at the cashiers' counter. 
[0011] The bar code system has so far been utilized 
only as a means for alleviating the burden of cashiers 
entering product identification numbers through key op- 
erations. As such, the code system has not been used 
for bypassing the chores of computer input operations 
such as menu selections when a desired program is to 
be activated selectively from among application pro- 
grams stored beforehand on a hard disk drive (HDD) of 
a general purpose personal computer. 

SUMMARY OF THE INVENTION 

[0012] Embodiments of the present invention seek to 
provide: an image processing apparatus; an image 
processing method; an image processing program stor- 
age medium; and a computer program product; for fa- 
cilitating diverse input operation such as selection of a 
menu and starting of an application program, wherein a 
user simply presenting a computer with a menu selec- 
tion card with a suitable two-dimensional code printed 
thereon causes the position of the code to be captured 
so that remote operations such as menu selection and 
application program activation are carried out according 
to information on the captured position. 
[001 3] According to one aspect of the present inven- 
tion, there is provided an image processing apparatus 
comprising: image acquiring means for acquiring an ob- 
ject image; displaying means for displaying the object 
image acquired by the image acquiring means; captured 
position information recognizing means for capturing a 
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predetermined image pattern from within the object im- 
age acquired by the image acquiring means in order to 
recognize position information about the image pattern 
relative to the object image; menu display controlling 
means for displaying a menu image in overlaid relation 
to the object image displayed by the displaying means 
in accordance with an outcome of the recognition by the 
captured position information recognizing means; and 
starting means for starting a previously related process 
based on the menu image displayed by the menu dis- 
play controlling means as well as on the position infor- 
mation recognized by the captured position information 
recognizing means. 

[0014] According to another aspect of the invention, 
there is provided an image processing method compris- 
ing the steps of: acquiring an object image; displaying 
the object image acquired in the image acquiring step; 
recognizing position information about a predetermined 
image pattern relative to the object image by capturing 
the image pattern from within the object image acquired 
in the image acquiring step; controlling display of a 
menu image in overlaid relation to the object image dis- 
played in the displaying step in accordance with an out- 
come of the recognition in the captured position infor- 
mation recognizing step; and starting a previously relat- 
ed process based on the menu image displayed by the 
menu display controlling step as well as on the position 
information recognized in the captured position informa- 
tion recognizing step. 

[001 5] According to a further aspect of the invention, 
there is provided an image processing program storage 
medium for storing an image processing program in a 
manner executable by a computer, the image process- 
ing program comprising the steps of: acquiring an object 
image; displaying the object image acquired in the im- 
age acquiring step; recognizing position information 
about a predetermined image pattern relative to the ob- 
ject image by capturing the image pattern from within 
the object image acquired in the image acquiring step; 
controlling display of a menu image in overlaid relation 
to the object image displayed in the displaying step in 
accordance with an outcome of the recognition in the 
captured position information recognizing step; and 
starting a previously related process based on the menu 
image displayed by the menu display controlling step as 
well as on the position information recognized in the 
captured position information recognizing step. 
[0016] Through the use of the inventive image 
processing apparatus, image processing method, and 
image processing program storage medium outlined 
above, a predetermined image pattern is first captured 
from an acquired object image. Position information 
about the captured image pattern relative to the object 
image is then recognized. A menu image is displayed 
on the basis of the captured position information, and a 
previously related process is started accordingly. 
[001 7] A better understanding of the invention will be- 
come apparent upon a reading of the following illustra- 



tive description with reference to the appended draw- 
ings, in which: 

BRIEF DESCRIPTION OF THE DRAWINGS 

5 

[0018] 

Fig. 1 is a schematic view showing the use of an 
illustrative a personal computer 1 to which the in- 
fo vention is applied; 

Fig. 2 is an explanatory view depicting specifica- 
tions of a two-dimensional code; 
Fig. 3 is a perspective view of an example of a port- 
able personal computer embodying the invention, 
*5 with its display part swung open away from its body; 
Fig. 4 is a plan view of the computer in Fig. 3; 
Fig. 5 is a left-hand side view of the computer in Fig. 
3 with its display part swung shut onto its body; 
Fig. 6 is a right-hand side view of the computer in 
20 Fig. 3 with its display part swung open 1 80 degrees 
relative to its body; 

Fig. 7 is a front view of the computer in Fig. 5; 
Fig. 8 is a bottom view of the computer in Fig. 6; 
Fig. 9 is a block diagram showing an internal struc- 
25 ture of the personal computer 1 in Fig. 3; 

Fig. 10 is a schematic view indicating a structure of 
an HDD 56 in Fig. 9; 

Fig. 1 1 is a flowchart of steps constituting a remote 
pointing process; 
30 Fig. 1 2 is a schematic view sketching a typical dis- 
play on an LCD 21 in Fig. 3; 
Fig. 1 3 is a schematic view depicting another typical 
display on the LCD 21 in Fig. 3; and 
Fig. 14 is a schematic view illustrating another typ- 
35 fcal display on the LCD 21 in Fig. 3. 

DETAILED DESCRIPTION OF ILLUSTRATIVE 
EMBODIMENTS 

40 [0019] Preferred illustrative embodiments of this in- 
vention will now be described with reference to the ac- 
companying drawings. 

[0020] Fig. 1 is a schematic view showing the use of 
an example of a personal computer 1 to which the in- 

45 vention is applied. In this example, a two-dimensional 
code 101 of a predetermined pattern is printed onto a 
menu selection card 100, or a label on which the two- 
dimensional code 1 01 is printed is attached to the card 
100. The two-dimensional code 101 may be simply re- 

50 ferred to as the 2D code hereunder where appropriate. 
[0021] AsshowninFig.2,the2Dcode101 constitutes 
a block unit-based rectangle measuring 9.5 blocks by 7 
blocks. The rectangle comprises a cell part A and a logo 
part B. in black, separated by a single block space. The 

5$ cell part A forms a square of 7 blocks per side compris- 
ing square cells arranged in a two-dimensional pattern. 
The logo part B includes a large-sized rectangle meas- 
uring 1 .5 blocks by 7 blocks. Illustratively, the logo part 
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B has a logo mark such as CyberCode (registered trade- 
mark) printed in blanks against a black background rep- 
resenting the code system name of the an code 1 01 . 
[0022] Illustratively, the following description is avail- 
able at the time of submitting this application from the s 
home page offered by this applicant at (URL): 
http://www.sony.co.jp/sd/PrcKJuctsPark/Consumer/ 
PCOM/PCG-C1 CAT/cybercode.html 
[0023] "What we call "CyberCode" is Sony's unique 
two-dimensional code system that offers about 16.77 10 
million different patterns (in 24 bits). Of these patterns, 
about one million patterns (in 20 bits) may be registered 
as desired for program start-up purposes. The remain- 
ing code patterns are reserved for future service expan- 
sion. "CyberCode" works as an index to what is repre- is 
sented by the code in question, the index allowing rele- 
vant information to be retrieved from computer storage. 
When a user starts a program through a new interface 
feature "CyberCode Finder", the user finds that the cor- 
responding information leaps from the object having the 20 
"CyberCode" onto the computer screen." 
[0024] The personal computer 1 is a notebook type 
computer comprising a CCD video camera 23 located 
in a display part 3. Illustratively, the personal computer 
1 recognizes a pattern of the 2D code 101 on the basis 25 
of image data about the object 100 and 2D code 101, 
the image data being acquired by the CCD video cam- 
era 23. The personal computer 1 performs a process in 
accordance with the pattern of the 2D code 101 thus 
obtained. so 
[0025] Figs. 3 through 8 depict structures of an exam- 
ple of a portable personal computer to which the inven- 
tion is applied. The personal computer 1 is a mini-note- 
book type personal computer that primarily comprises 
a body 2 and a display part 3 attached swingingly to the 3S 
body 2. Fig. 3 is a perspective view of the computer with 
the display part 3 swung open away from the body 2. 
Fig. 4 is a plan view of the computer in Fig. 3. Fig. 5 is 
a left-hand side view of the computer with the display 
part 3 swung shut onto the body 2. Fig. 6 is a right-band 40 
side view of the computer with the display part 3 swung 
open 1 80 degrees relative to the body 2. Fig. 7 is a front 
view of the computer in Fig. 5. Fig. 8 is a bottom view of 
the computer in Fig. 6. 

[0026] The face of the body 2 comprises a keyboard 45 
4 and a stick type pointing device 5. The keyboard 4 is 
used to input characters, symbols, etc., and the stick 
type pointing device 5 is operated to move a mouse cur- 
sor. Also furnished on the body face are a speaker 8 for 
sound output and a shutter button 10 operated to take &> 
a picture using the CCD video camera 23 mounted on 
the display part 3. 

[00271 A pawl 1 3 is provided at the upper end of the 
display part 3. As shown in Fig. 5, with the display part 
3 swung closed onto the body 2, the pawl 1 3 hooks on 55 
to a hole 6 in the body 2. At the front of the body 2 is a 
slide lever 7 furnished in a crosswise movable fashion. 
The slide lever 7 is used to lock and unlock the pawl 1 3 



so that the pawl 1 3 is engaged with and disengaged 
from the hole 6. With the pawl 1 3 unlocked, the display 
part 3 may be swung open away from the body 2. Adja- 
cent to the pawl 1 3 is a microphone 24 which, as depict- 
ed in Fig. 8, may pick up sound not only from the front 
but also from the back side of the body 2. 
[0028] The front of the body 2 further comprises a pro- 
grammable power key (PPK) 9. An air outlet 11 is pro- 
vided on the right-hand side of the body 2, as shown in 
Fig. 6. At the lower end in front of the body 2 is an air 
inlet 1 4 as depicted in Fig. 7. To the right of the air outlet 
11 is a slot 1 2 that accommodates a PCMCIA (Personal 
computer Memory Card International Association) card 
(called a PC card). 

[0029] An LCD (liquid crystal display) 21 for displaying 
images is provided on the front of the display part 3. At 
the upper end of the LCD 21 is an image pickup part 22 
mounted rotatably on the display part 3. More specifi- 
cally, the image pickup part 22 is rotatable to any posi- 
tion within a range of 180 degrees in the same direction 
as the LCD 21 and in the opposite direction thereof (i. 
e., toward the back). The image pickup part 22 is fur- 
nished with the CCD video camera 23. 
[0030] At the lower end of the display part 3 on the 
body side is a group of lamps including a power lamp 
PL, a battery lamp BL, a message lamp ML and other 
LEDs. Reference numeral 40 in Fig. 5 denotes a power 
switch furnished on the left-hand side of the body 2, and 
reference numeral 25 in Fig. 7 represents an adjusting 
ring used to adjust the focus of the CCD video camera 
23. Reference numeral 26 in Fig. 8 stands for a cover 
that conceals an opening through which to install an ad- 
ditional memory into the body 2, and reference numeral 
41 denotes a hole through which to insert a pin to unlock 
the cover 26. 

[0031] Fig. 9 illustrates an internal structure of the per- 
sonal computer 1 . An internal bus 51 is connected to a 
CPU (central processing unit) 52, a PC card 53 inserted 
as needed, a RAM (random access memory) 54, and a 
graphic chip 81. The internal bus 51 is coupled to an 
external bus 55. The external bus 55, for its part, is con- 
nected to a hard disk drive (HDD) 56. an I/O (input/out- 
put) controller 57, a keyboard controller 58, a stick type 
pointing device controller 59, a sound chip 60, an LCD 
controller 83, and a modem 50. 
[0032] The CPU 52 is a controller that controls diverse 
computer functions. The PC card 53 is installed as need- 
ed when an optional function is to be added. 
[0033] Image data captured by the CCD video camera 
23 are forwarded to a processing part 82 for processing. 
The image data processed by the processing part 82 
are input to the graphic chip 81. The graphic chip 81 
stores the input video data into an internal vRAM 81 A, 
and retrieves the data from the memory as needed for 
output to the LCD controller 83. Given the image data 
from the graphic chip 81 , the LCD controller 83 outputs 
the data to the LCD 21 for display. Back lights 84 are 
provided to illuminate the LCD 21 from the back. 
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[0034] When the personal computer 1 is booted up, 
an electronic mail program (an application program) 
54A, an auto pilot program (another application pro- 
gram) 54B and the OS (operating program) 54C are 
transferred from the HDD 56 to the RAM 54 and retained s 
therein. 

[0035] The electronic mail program 54 A is a program 
that exchanges communication messages with an ex- 
ternal entity using a communication line such as a tele- 
phone line and by way of a network. A received mail 10 
acquisition function is specifically included in the elec- 
tronic mail program 54A. The received mail acquisition 
function checks a mail server 93 to see if a mail box 93A 
therein contains any mail addressed to this program (i. 
e. , to the user). If any such mail is found in the mail box 15 
93A, the received mail acquisition function carries out a 
suitable process to acquire that mail. 
[0036] The auto pilot program 54B is a program that 
starts up and carries out a plurality of predetermined 
processes (or programs) in a predetermined sequence. 20 
[0037] The OS (operating system) 54C controls basic 
computer functions. Typical operating systems are Win- 
dows 95 (registered trademark), Windows 98 (regis- 
tered trademarks), and the like. 

[0038] Meanwhile, as shown in Fig. 10, the hard disk 25 
drive (HDD) 56 connected to the external buc 55 con- 
tains the electronic mail program 56A, auto pilot pro- 
gram 56B, OS (operating system) 56C, a two-dimen- 
sional code database (called the 2D code database 
hereunder) 56D, a finder application program (called the so 
FA program hereunder) 56E, N two-dimensional code 
associated application programs (called the 2D code 
programs hereunder) 56F-1 through 56F-N (the pro- 
grams may be generically referred to as the 2D code 
program 56F if distinction between the individual pro- 35 
grams is unnecessary), and a 2D code program menu 
information database (called the menu information da- 
tabase hereunder) 56G. 

[0039] The 2D code database 56D stores identifica- 
tion numbers of two-dimensional codes (called 2D code 40 
IDs) attached to previously registered menu selection 
cards, such as 2D code IDs of the 2D codes 101 . 
[0040] The FA program 56E illustratively carries out a 
two-dimensional code extracting process whereby im- 
age data representing a given two-dimensional code are 45 
extracted from the image data held in the VRAM 81 A of 
the graphic chip 81 . The FA program 56E also performs 
a remote pointing process (described later) to start the 
corresponding 2D code program 56F accordingly. The 
FA program 56E has an API (application programming 50 
interface) that is used to call the 2D code program 56F 
and to exchange various messages. 
[0041] The 2D code programs 56F-1 through 56F-N 
are application programs that are started by the FA pro- 
gram 56E. The 2D code programs 56F are capable of 55 
exchanging data with the FA program 56E through the 
API of the latter. Among these 2D code programs 56F, 
the program 56F-1 comprises a menu selection pro- 



gram for displaying menu selection buttons and a point- 
ing cursor, to be described later. 
[0042] The menu information database 56G holds in- 
formation about positions (regions) of displayed menu 
selection buttons. The database 56G also retains pro- 
gram IDs of the 2D code programs 56F to be started 
corresponding to a selected menu selection button, as 
well as programs IDs identifying general application pro- 
grams such as a word processor. 
[0043] During the booting process, the OS 56C, auto 
pilot program 56B and electronic mail program 56 A are 
transferred successively from the hard disk drive 56 to 
the RAM 54 and stored in that memory. 
[0044] Returning to Fig. 9, the I/O controller 57 has a 
microcontroller 61 equipped with an I/O interface 62. 
The microcontroller 61 is constituted by the I/O interface 
62, a CPU 63, a RAM 64 and a ROM 69 which are in- 
terconnected. The RAM 64 includes a key input status 
register 65, an LED (light-emitting diode) control register 
66, a set time register 67, and a register 68. The set time 
register 67 is used to start the operation of a start se- 
quence controller 76 when a time preset by the user (i. 
e:; starting condition) is reached. The register 68 holds 
a correspondence between a preset combination of op- 
eration keys (starting condition) on the one hand and an 
application program to be started on the other hand, 
when the user inputs the preset combination of opera- 
tion keys, the corresponding application program (e.g., 
electronic mail program) is started. 
[0045] When the fingertip-operated programmable 
power key (PPK) 9 is pushed, the key input status reg- 
ister 65 gets and retains an operation key flag. The LED 
control register 66 is used to control the illumination of 
the message lamp ML indicating that boot-up status of 
an application program (e.g., electronic mail program) 
which is held in the register 68. A desired time of day 
may be set to the set time register 67. 
[0046] The microcontroller 61 is connected to a back- 
up battery 74. The battery 74 allows contents ol the reg- 
isters 65, 66 and 67 to be retained when power to the 
body 2 is turned off. 

[0047] The ROM 69 in the microcontroller 61 contains 
in advance a wake-up program 70, a key input monitor- 
ing program 71 and an LED control program 72. The 
ROM 69 is illustratively composed of an EEPROM (elec- 
trically erasable and programmable read only memory). 
The EEPROM is also called a flash memory. The micro- 
controller 61 is connected to an RTC (real-time clock) 
75 that keeps the current time. 
[0048] The wake-up program 70 in the ROM 69 is a 
program that checks to see if a preset time in the set 
time register 67 is reached on the basis of timeof-day 
data from the RTC 75. When the preset time is reached, 
the wake-up program 70 starts up a predetermined 
process (or program). The key input monitoring program 
71 continuously monitors whether the PPK 9 is pushed 
by the user. The LED control program 72 controls the 
lighting of the message lamp ML. 
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[0049] Furthermore, the ROM 69 contains a BIOS 
(basic input/output system) 73. The BIOS is a software 
program that controls exchanges of data (input and out- 
put) between the OS or application software on the one 
hand and peripheral devices (e.g., display part, key- s 
board, hard disk drive) on the other hand. 
[0050] The keyboard controller 58 connected to the 
external bus 55 controls input from the keyboard 4. The 
stick type pointing device controller 59 controls input 
from the stick type pointing device 5. 10 
[0051] The sound chip 60 receives input from the mi- 
crophone 24, and supplies sound signals to the built-in 
speaker 8. 

[0052] The modem 50 permits connection to a com- 
munication network such as the Internet 92 and to the is 
mail server 93 through a public telephone line 90 and 
an Internet service provider 91 . 
[0053] The power switch 40 is operated to tu rn on and 
off the power supply. A half-push switch 85 is activated 
when the shutter button 10 is half -pushed. A full-push 20 
switch 86 is turned on when the shutter button 10 is fully 
pushed. A reverse switch 87 is turned on when the im- 
age pickup part 22 is rotated by 180 degrees (i.e., when 
the CCD video camera 23 is rotated into a direction suit- 
able for picking up an image on the opposite side of the 2s 
LCD 21). 

[0054] Described below with reference to the flow- 
chart of Fig. 11 are steps in which the FA program 56E 
carries out the remote pointing process in conjunction 
with the 2D code program 56F-1 (menu selection pro- 30 
gram). 

[0055] The FA program 56E is first started from the 
HDD 56. In step S1, the FA program 56E executed by 
the CPU 52 acquires an image captured by the CCD 
video camera 23. 1 n step S2, the FA program 56E caus- 35 
es the processing part 82 to process data representing 
the captured image, writes the processed image data to 
the VRAM 81 A of the graphic chip 81, and controls the 
LCD controller 83 so as to display the captured image 
on a finder screen 201 of the LCD 21 as shown in Fig. 40 
12. In the description that followo, the whole process in 
which the FA program 56E causes the LCD controller 
83 to display an image on the finder screen 201 of the 
LCD 21 will be simply described as the FA program 56E 
itself displaying the image on the finder screen 201 . 45 
[0056] In step S3, the FA program 56E extracts and 
recognizes a 2D code 101 from the image data repre- 
senting the object image acquired by the CCD video 
camera 23 in step S1 . At the same time, the FA program 
56E recognizes position information on the 2D code 1 01 so 
relative to the object image displayed on the finder 
screen 201 (e.g., central point of the 2D code 101). In 
the example of Fig. 12, a 2D code 101 is extracted from 
an image of a menu selection card 100 captured by the 
CCD video camera 23. The position and a 2D code ID ss 
of the 2D code 101 in question are captured, and infor- 
mation about the captured position is acquired. After the 
FA program 56E has recognized the 2D code ID of the 



2D code 1 01 and acquired its captured position infor- 
mation in step S3, step s4 is reached. 
[0057] In step S4, a search is made through the 2D 
code database 56D to see if the 2D code ID recognized 
in step S3 matches a 2D code established beforehand 
in connection with the menu selection program (2D code 
program 56F-1 ). If the 2D code ID in question is judged 
in step S4 to match the predetermined 2D code, step 
S5 is reached. 

[0058] In step S5, a check is made to see if the menu 
selection program (2D code program 56F-1 ) is being ac- 
tive. If the program is not judged to be active, step S6 
is reached. 

[0059] In step S6, the menu selection program (2D 
code program 56F-1) is started. Then the menu selec- 
tion program executed by the CPU 52 searches the 
menu information database 56G for a menu image com- 
prising suitable selection buttons, and displays the re- 
trieved menu image in overlaid relation to the captured 
image displayed in step S2. Illustratively, the menu im- 
age appears on the finder screen 201 as shown in Fig. 
13. Information about regions representing four menu 
selection buttons 300-1 through 300-4 is retrieved from 
the menu information database 56G, and the buttons 
are displayed on the basis or the retrieved region infor- 
mation. 

[0060] Step S6 is followed by step S7 in which the FA 
program 56E receives the relevant 2D code ID from the 
menu. detection program thus started, step S7 is fol- 
lowed by step S3. 

[0061] If in step S5 the menu selection program is 
judged to be already active, step S8 is reached. In step 
S8, the menu selection program (2D code program 56F- 
1) displays a pointing cursor (shown in Fig. 13; simply 
called the cursor hereunder) 301 corresponding to the 
captured position information on the 2D code 101 rec- 
ognized in step S3. In the example of Fig. 1 3, the cursor 
301 has a point A located at the center of the 2D code 
101, the point A being encircled by two concentric cir- 
cles. Two lines are shown crossing the point A, one par- 
alleling the X-axis of the finder screen 201 , the other in 
parallel with the Y-axis of the screen 201 . 
[0062] In step S9, the menu selection program (2D 
code program 56F-1 ) checks to see if the position point- 
ed to by the cursor 301 in step S8 (e.g., point A of the 
cursor 301 ) points to any one of the menu selection but- 
tons 300-1 through 300-4. If the point A of the cursor 
301 is judged to be positioned in a region of the menu 
selection button 300-i (i - 1 in the example of Fig. 1 3) as 
shown in Fig. 13, step S10 is reached. For the descrip- 
tion that follows, it is assumed that the point A of the 
cursor 301 points to the menu selection button 300-i as 
illustrated in Fig. 13. 

[0063] In step S10, the menu selection program (2D 
code program 56F-1 ) permits display of only the menu 
selection button 300-1 pointed to by the cursor 301 (i. 
e., the button highlighted) as shown in Fig. 14. The user 
is thus informed of the current selection of the menu se- 
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lection button 300-1. 

[0064] In step S11, the menu selection program (2D 
code program 56F-1) counts the time during which the 
cursor 301 is pointing to the menu selection button 300-i. 
If the menu selection button 300-i is judged to have been 
pointed to in excess of a predetermined period of time, 
the program goes to step S1 2. 
[0065] Step S12 can be reached solely upon elapse 
of at least a predetermined period of time during which 
the cursor 301 has kept pointing to the menu selection 
button 300-i. This arrangement prevents a subsequent 
process from being carried out inadvertently with the 
cursor judged to be pointing to a different menu selec- 
tion button 300-i while the presented position of the 
menu selection card 1 00 has yet to be fixed. 
[0066] If in step S11 the cursor 301 is judged to have 
pointed to the menu selection button 300-i in excess of 
a predetermined period of time, the menu selection pro- 
gram (2D code program 56F-1) goes to step S12 in 
which the program searches the menu information da- 
tabase 56G for a program ID held corresponding to the 
region information about the menu selection button 
300-i pointed to by the cursor 301. when a program ID 
is retrieved, the 2D code program 56F or a general ap- 
plication program such as a word processor corre- 
sponding to the retrieved program ID is started. There- 
after the process of Fig. 11 is terminated. 
[0067] If in step 54 the FA program 56E, following the 
search through the 2D code database 56D, judges that 
the 2D code ID recognized in step S3 does not corre- 
spond to the previously established 2D code, then step 
S13 is reached. In step S13, a 2D code program 56F 
other than the menu selection program or a general ap- 
plication program such as a word processor having the 
program ID in question is started. Thereafter the proc- 
ess of Fig. 11 is terminated. 

[0068] If in step S9 the cursor 301 is not judged to be 
pointing to the menu selection button 300-i, or if the cur- 
sor 301 is not judged in step S11 to have pointed to the 
menu selection button 300-i in excess of a predeter- 
mined period of time, then step S3 is reached again. 
Step S3 and subsequent steps are then repeated. 
[0069] As described, when the cursor 301 at the rel- 
evant displayed position of a 2D code 101 is arranged 
to point to a menu selection button 300 for a certain pe- 
riod of time, the process corresponding to the button 300 
is carried out without operation of the keyboard or 
mouse at all. 

[0070] A computer program designed to perform the 
above-described processes may be retained on such 
package media as floppy disks, CD-FtOMs and DUDS; 
on semiconductor memories, magnetic disks and the 
like where the program is stored temporarily or perma- 
nently; on wired and wireless communication media 
such as local area networks, the Internet, digital satellite 
broadcasting networks; or in diverse communication in- 
terfaces such as routers and modems for transmitting 
or receiving the program offered by the foregoing media. 



Such media, networks, interfaces and other measures 
allow the program to be installed in computers for pro- 
gram execution. The image processing program stor- 
age medium as mentioned in this specification refers 
5 broadly to all such media, networks, interfaces and 
measures. 

[0071] According to the illustrative image processing 
apparatus, image processing method, image process- 
ing program storage medium and computer program 

io product as described above, a predetermined image 
pattern is first captured from an acquired object image. 
Position information about the captured image pattern 
relative to the object image is then recognized. A menu 
image is displayed on the basis of the captured position 

is information, and a previously related process is started 
accordingly. Specifically, the user's simple action of pre- 
senting a computer with a menu selection card with a 
suitable 2D code printed thereon causes the position of 
the 2D code to be captured. The captured position in- 

20 formation is used as a basis for effecting remote oper- 
ations such as menu selection and starting of an appli- 
cation program. This provides an easy-to-operate plat- 
form eliminating the need for complicated input opera- 
tions done conventionally using the mouse and other in- 

2S put device. 

[0072] As many apparently different embodiments of 
this invention may be made without departing from the 
spirit and scope thereof, it is to be understood that the 
invention is not limited to the specific embodiments de- 

30 scribed hereinabove. 



Claims 

35 1. An image processing apparatus comprising: 

image acquiring means for acquiring an object 
image; 

displaying means for displaying said object im- 
40 age acquired by said image acquiring means; 

captured position information recognizing 
means for capturing a predetermined image 
pattern from within said object image acquired 
by said image acquiring means in order to rec- 
45 ognize position information about said image 

pattern relative to said object image; 
menu display controlling means for displaying 
a menu image in overlaid relation to said object 
image displayed by said displaying means in 
50 accordance with an outcome of the recognition 

by said captured position information recogniz- 
ing means; and 

starting means for starting a previously related 
process based on said menu image displayed 
55 by said menu display controlling means as well 

as on said position information recognized by 
said captured position information recognizing 
means. 
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acquiring an object image; 
displaying said object image acquired in said 
image acquiring step; 

recognizing position information about a prede- 
5 termined image pattern relative to said object 

image by capturing said image pattern from 
within said object image acquired in said image 
acquiring step; 

controlling display of a menu image in overlaid 
io relation to said object image displayed in said 

displaying step in accordance with an outcome 
of the recognition in said captured position in- 
formation recognizing step; and 
starting a previously related process based on 
15 said menu image displayed by said menu dis- 

play controlling step as well as on said position 
information recognized in said captured posi- 
tion information recognizing step. 



2. An image processing apparatus according to claim 
1 , wherein said captured position information rec- 
ognizing means captures a visible code from within 
said object image acquired by said image acquiring 
means, and recognizes position information about 
an image pattern of said visible code relative to said 
object image. 

3. An image processing apparatus according to claim 
1, wherein said captured position information rec- 
ognizing means captures a two-dimensional code 
attached to an object from within said object image 
acquired by said image acquiring means, and rec- 
ognizes position information about an image pat- 
tern of said two-dimensional code relative to said 
object image. 

4. An image processing apparatus according to claim 
1 , wherein said menu display controlling means dis- 
plays selection buttons as said menu image, and 20 
displays a pointing cursor in a position correspond- 
ing to said position information recognized by said 
captured position information recognizing means. 

5. An image processing apparatus according to claim 
4, wherein, if said pointing cursor is displayed in ex- 
cess of a predetermined period of time inside a spe- 
cific selection button region displayed by said menu 
display controlling means, then said starting means 
starts a process previously related to said specific 
selection button. 

6. An image processing method comprising the steps 
of: 

acquiring an object image; 
displaying said object image acquired in said 
image acquiring step; 

recognizing position information about a prede- 
termined image pattern relative to said object 
image by capturing said image pattern from 
within said object image acquired in said image 
acquiring step; 

controlling display of a menu image in overlaid 
relation to said object image displayed in said 45 
displaying step in accordance with an outcome 
of the recognition in said captured position in- 
formation recognizing step; and 
starting a previously related process based on 
said menu image displayed by said menu dis- so 
play controlling step as well as on said position 
information recognized in said captured posi- 
tion information recognizing step. 

7. An image processing program storage medium for 55 
storing an image processing program in a manner 
executable by a computer, said image processing 
program comprising the steps of: 



8. A computer program product containing an image 
processing program comprising the steps of: 



acquiring an object image; 
displaying said object image acquired in said 
25 image acquiring step; 

recognizing position information about a prede- 
termined image pattern relative to said object 
image by capturing said image pattern from 
within said object image acquired in said image 
30 acquiring step; 

controlling display of a menu image in overlaid 
relation to said object image displayed in said 
displaying step in accordance with an outcome 
of the recognition in said captured position in- 
35 formation recognizing step; and 

starting a previously related process based on 
said menu image displayed by said menu dis- 
play controlling step as well as on said position 
information recognized in said captured posi- 
40 tion information recognizing step. 
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FIG. 11 
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